1. Technical Field
Present invention embodiments relate to testing jobs in a computing environment, and, more specifically, to a framework for developing, deploying, and executing tests of Extract, Transform, and Load jobs.
2. Discussion of the Related Art
Many organizations use Extract, Transform, and Load (ETL) jobs with a large number of stages to integrate information on a regular basis. These jobs are generally designed using a graphical interface, sometimes referred to as a canvas. The designer of an ETL job drags and drops stages on the canvas and creates connections (sometimes referred to as links) between the stages. The stages and connections between them are treated as interconnected nodes in a graph. Code is generated for the job based on the graph. A number of optimization steps are performed on the generated code to produce a final executable that is ultimately deployed and used.
At various times after a job has been produced, a user may want to test the job to determine whether errors have arisen. This is referred to as regression testing. For example, a user might perform regression testing after (1) moving a job from the development phase to a quality assurance phase or from quality assurance to production; (2) changing the ETL system, e.g. applying a software fix; (3) migrating a job from one system to another, e.g. to new hardware; or (4) migrating a job from one version of an ETL product to the next version of the product. Typical ETL test and production environments place restrictions on applying changes to existing compiled/executable jobs, which requires that testing related considerations must either be built into the job during development or performed manually against the executable on an ad hoc basis.